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2 LISTENING ASSESSMENT VIA THE 

COMMUNICATION COMPETENCY ASSESSMENT INSTRUMENT 



Rebecca B. Rubin a Pat Shepherd 
Kent State University 



Abstract 



This paper examines the development of the listening test portion 
of the Communication Competency Assessment Instrument and reports on a 
new method of achieving listening test scores. The multiple choice 
method was developed during the Summer and Fall of 1985 and the results 
of this Instrument were compared to the Items In the CCAI, and the 
subjects' scores on the Kentucky Comprehensive Listening Test and the 
Watson-Barker Listening Test. Results Indicate that the open-ended 
version of the CCAI Is superior to a multiple choice format. 
Implications of testing listening ability are discussed. 
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LISTENING ASSESSMENT VIA THE 
COMMUNICATION COMPETENCY ASSESSMENT INSTRUMENT 



The Communication Competency Assessment Instrument was developed as 
a college-level comprehensive communication competence measure. The 
goal of the instrument 1s to identify students who may have difficulties 
with both sending and receiving communication 1n an educational setting 
(Rubin, 1982b). Thus, the comprehensive nature of the Instrument 
mandated that listening, as well as speaking and Interpersonal skills, 
be assessed. 

The listening portion of the CCAI requires that students view a 
videotaped representation of a first day 1n a listening class. The 
instructor of this class first explains the course requirements, 
explains three factors which affect effective listening, gives three 
ways 1n which students can Improve their listening abilities, and 
explains the first assignment for the class. All this 1s accomplished 
1n the 7 minute and 40 second segment. The videotape was kept to this 
size so that long-term memory would not come Into play when listening 
was assessed. When the instrument was first belno developed, no other 
videotaped listening tests were available for use. It was felt that a 
videotaped version of the stimulus material would be superior to an 
audiotaped version since (1) picture and sound 1s more typical of the 
every-day listening students experience 1n classroom settings, and (2) 
the dual -channel representation would provide more information for 
students and enhance recall and learning (Nelson A Moll, 1951; Gunter, 



To determine concurrent validity of the listening portion of this 
Instrument, 30 students viewed the videotape and completed 1n writing 
the four items which are assessed via the listening tape. At the same 
time, they completed the L1eb-Brilhart (1965) listening test. Scores on 
the two Instruments correlated at .69 (p < .0001) (Rubin, 1982b), 
indicating concurrent validity. 

Normally, however, 1n keeping with the nature of the measurement 
mode (oral), students are orally asked the four questions about the 
listening videotape and they respond orally to these questions. Two 
earlier studies of the questions students are asked about the videotape 
(Rubin, 1982b, 1985) reported that, on a 5-po1nt scale, the mean scores 
for differentiating fact from opinion were 3.57 and 3.49, for 
understanding the main points 1n the stimulus lecture, 3.06 and 2.73, 
for Identifying the class assignment, 3.5 and 3.17, and for summarizing 
Instructions to others, 3.49 and 3.02, respectively. Henzl, Mabry and 
Powell (1983) reported that the last three of these four Hams factored 
Into a listening factor when the number of factors was undetermined by 
the researchers and when Henzl (personal communication) forced the 19 
Items of the CCAI Into 4 factors, all four Items comprised a listening 
factor. Therefore, for this current study, 1t was determined that the 
four Items would be considered together as the listening portion of the 
CCAI and to differentiate this from the second method of testing 
listening using the CCAI, 1t would be designated the CCAI-OE (open- 
ended) scale. 
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The oral response mode was and 1s seen as the Ideal format for 
assessing listening for three basic reasons. First, havlno the rater 
ask the questions requires the student to listen to the question in 
addition to the stimulus. To be certain that students understand what 
is asked of them, raters can look for puzzlement and can elaborate on or 
repeat missed portions of the question. This gives tho student the best 
chance for success. 

Second, reading ability is not an intervening variable when 
li "enlng ability is assessed. We have noticed, when using other 
liscenlng tests where a specified amount of time 1s given to the student 
to read and respond to the question, some students are not able to 
complete reading the question before the next listening stimulus 
material begins. Thus, their listening scores are partly based on 
reading ability. 

Third, students have time to recall what was presented 1n the 
videotape without time pressure. The pressure of moving on to the next 
question may force Incorrect answers; having time to think about what 
was heard may increase the chances for correct responses. Again, the 
goal of this Instrument is to Identify students' communication problem 
areas, but under conditions where they are given every opportunity for 
success. rr 

The main criticism of the oral, open-ended approach 1s that 1t Is 
not practical for large-scale testing situations and, therefore, not 
cost-effective. Students must be tested Individually. Also, all four 
of the listening assessment items (fact-op1n1on distinction, 
understanding suggestions, identifying class assignments, and 
summarizing) are not typical of most listening tests. Two of these 
(fact-op1n1on and summarizing) Involve other oral communication skills. 
Therefore, there are only two Items which assess listening 1n Its pure 
sense. Some users of the CCAI have suggested that an additional measure 
be created that Includes multiple items so that (1) the listening 
videotape could be used as a stand-alone listening test or (2) as an 
additional measure of listening ability. 

To create such an instrument, we kept 1n mind the concerns we 
Identified previously (about time pressure, making sure the student 
understands the questions, and reading ability). We determined that the 
instrument should be completed only after students listened to the 
videotape, that students should receive as much time as they need to 
answer the questions, and that the items should be multiple-choice 1n 
nature (to Increase understanding). 

During the summer of 1985, we created 12 multiple-choice questions 
about the videotaped material and pretested these with a group of 15 
students enrolled 1n the basic communication class at a regional campus 
of Kent State University. These students (10 females-mean age of 26.5 
years— and 5 males— mean age of 21.2 years) viewed the videotape and 
were asked to answer the questions and comment on any questions which 
were difficult to understand or vague. These students found no problems 
in how the questions were phrased and Indicated that the questions 
appeared to test what they had heard via the videotape. 
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As we analyzed the results, two questions which all students 
answered correctly were eliminated from this version of the test and 
some of the response items were refined for the next version of the 
CCAI-MC (mult'ple-choice form. This form was then administered to a 
large group of students aiong with two other listening assessment 
instruments. 



METHOD 

Sample 

Three hundred and thirty nine students enrolled in introductory 
speech communication courses at Kent state University were asked to 
complete three listening assessment instruments. Two hundred and 
seventy-eight of these students were enrolled in a basic public speaking 
course and 54 were enrolled in sophomore- level communication theory 
course. 

Demographic data collected on the subjects revealed that the mean 
age of this group was 20.98 years, students had completed an average of 
47.61 college credits, and that 46.6% were female and 52.2% were male. 



Procedures 

Students were asked to complete three listening measures: The 
Kentucky Comprehensive Listening Test (Bostrom & Waldhart, 1980b), the 
Watson-Barker Listening Test (rorm B) (Watson & Barker, 1983) and the 
two forms of the CCAI (-0E and -MC). From a random starting point, the 
three measures were presented to students on three separate days within 
a one week period in a counterbalanced manner so that order effects 
would be eliminated. However, because of circumstances surrounding the 
use of videotape in one large lecture hall, the six order groups were 
not equal in size. 

The CCAI-OE consisted of the four questions that are orally asked 
to students during the typical CCAI testing session. They were 
presented in writing first to the students so that students would be 
recalling the information that they had listened to on the videotape. 
As each student completed this form, ** was collected and the student 
was given the multiple-choice form. 

The open-ended written CCAI responses were coded on a scale which 
ranges from 5 (the most appropriate response) to 1 (least appropriate). 
This procedure and anchors for each of the 5 points are detailed in the 
CCAI rating book (Rubin, 1982a). Both authors coded these responses 
independently and inconsistencies were resolved through discussion. The 
total score (of the 4 items) constituted the measure of CCAI-OE 
listening. The multiple choice items in the CCAI-MC were scored 
according to the number of correct responses that students obtained. 
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Four components of listening are measured by the Kentucky 
Comprehensive Listening Test (KCLT): Short Term Listening (STL), Short 
Term Listening with Rehearsal (STL-R), INTERpretation of meaning 
(INTER), and LECTure comprehension (LECT). By combining six items of 
the STL section and six items in the STL-R section, a DISTraction (DIST) 
score is computed. These five scores are combined and are provided to 
listeners in the form of a listening profile. In the past, Bostrom ana 
Waldhart (1980a) found that short term listening was closely related to 
oral performance and lecture comprehension was related to general mental 
ability and written examination scores. Based on these findings, we 
predicted that the CCAI-OE and CCAJ-MC would be more strongly related to 
LECT scores than to the other components of the KCLT. 

Since the KCLT test was administered in a large lecture hall to 235 
of the subjects, it was impossible to police students during this test. 
The last section of the test asks students not to look at the questions 
until the listening stimulus is completed. However, we noted that many 
students were filling out the questions as they listened to the test. 
In their next testing session we explained to the students that this 
would damage the results of the study and asked them to place an 
asterisk by their name if they completed the answers as they were 
listening during the KCLT. Those students who complied with our request 
(N = 12) were eliminated from analysis on this measure. 

The Watson-Barker Listening Test (WBLT) is divided into five parts, 
each designed to test a particular type of listening skill. Part 1 
measures the listener's skill in INTERpreting message content in Short 
Term listening (INTER-ST). Part II measures the listener's ability to 
evaluate DIALOGue/conversations and interpret the meanings implied by 
the speakers (DIALOG), part III requires listeners to answer questions 
after listening to a series of SHort LECTures (SH-LECT). In Part IV, 
questions and statements are used to measure the listener's skill in 
interpreting how something is said or the EMOTion being communicated 
rather than tfie~content of the message (EMOT). And Part V measures a 
listener's ability to follow DIRECTions and instructions (DIRECT). 

Based on the goals of the five parts of the Watson-Barker test, we 
predicted that the CCAI-OE and CCAI-MC scores would correlate most 
highly with the SH-LECT and DIRECT components of the test and the SH- 
LECT component would be related to the LECT component of the KCLT. In 
addition, following from the findings of Applegate and Campbell (1985), 
we predicted high correlations between the STL-R component of the KCLT 
and the INTER-ST component of the WBLT, between the KCLT's STL component 
and the WBLT's SH-LECT component, between the KCLT's LECT component and 
the WBLT s DIRECT component, and between the totalled scores of the WBLT 
and KCLT. 

The results from the three listening measures were returned to the 
students as a listening profile. Students' grades were unaffected by 
their scores on these instruments. Since these instruments were admin- 
istered on different days, missing data occurred. In all statistical 
analyses, case-wise deletion was used when comparisons were made among 
the various listening tests. Since students had indicated their sex on 
the answer sheets, we included sex as a variable in the data analyses. 
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RESULTS 

Mean, median, mode, and standard deviations for all the instruments 
are presented in Table 1. As predicted, and as demonstrated in Table 2, 
the CCAI-OE (r = .17, p < .01) and CCAI-MC (r = .21, p < .001) total 
scores were more strongly related to lecture comprehension (LECT) score 
in the Kentucky Comprehensive Listening Test than to any of the other 
components. As Table 2 also indicates, the other components of the KCLT 
were virtually unrelated to the items assessed in the CCAI. 

We also predicted that the CCAI-OE and CCAI-MC scores would 
correlate highly with the SH-LECT and DIRECT components of the Watson- 
Barker test. As Table 2 indicates, the SH-LECT component correlated 
at .31 (p < .001) with the CCAI-OE and at .29 (p < .001) with the CCAI- 
MC. Three of the four questions in the CCAI -Ot (with the exception of 
the fact/opinion distinction) were correlated with this item. The 
correlations between the two CCAI forms with the DIRECT component were 
smaller, yet statistically significant. Once again, all questions 
except the fact/opinion Item were significantly correlated with the 
DIRECT component. In general , al 1 the Watson-Barker components were 
positively and statistically significantly related to the two CCAI 
forms. 

Applegate and Campbell's (1985) results led us to believe that the 
WBLT SH-LECT component would be related to the LECT component of the 
KCLT. These two components did correlate at .17 (p < .01). In 
addition, we predicted high correlations between the STL-R component of 
the KCLT and the INTER-ST component of the WBLT (r = .13, p < .05), 
between the KCLT's STL component and the WBLT's SH-LECT (r = .10, n.s.) 
component, between the KCLT's LECT component and the WBLT's" DIRECT 
component (r = .18, p < .01), and between the totalled scores of the 
WBLT and KCLT (r = 32, £ < .001). Other strong relationships existed 
between the two instruments. In particular, the WBLT's INTER-ST 
correlated significantly with the KCLT's STL (r = .19, p < .01) and LECT 
(r = .24, £ < .001) components. And the WBLT'slMOT component 
correlated with the KCLT's INTER component (r = .23, £< .001). 



DISCUSSION 

As the findings seem to indicate, there are many similarities 
between the Kentucky and Watson-Barker tests. In addition, the CCAI 
forms seem to be related more strongly to the Watson-Barker test than to 
the Kentucky test. The strongest relationship among the listening tests 
exists between the WBLT and the CCAI-OE. This lends support for the 
concurrent validity of both of these instruments. It 1s, however 
explained by the fact that the two instruments measure similar listening 
components; both measure listening and directions. 
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Second, the CCAI-OE form is superior to the CCAI-MC form. Even 
though many seek a quick and easy format for rating listening ability, 
the open-ended responses seem superior (in the form of concurrent 
validity) to the multiple-choice format. Ease of guessing on a 
multiple-choice test is most probably why the relationship between the 
open-ended anc multiple-choice formats were not more closely related. 

In creating the multiple choice version, we realized that there are 
two main issues that listening-test developers must consider. What is 
it that is or should be tested and what mode of testing is best. — 
Through our experiences in comparing these three instruments we have 
developed some observations on the assessment process: 

1. Understanding of concepts presented in a lecture format is an 
important part of a college student's listening environment. 
However, real -life situations differ from the listening test 
situations we have all created. The major difference is that 
students are allowed (and encouraged) to take notes during lectures. 
A lecture without note-taking is an unusual situation. Thus, the 
material presented in the listening stimulus tape .iust be 
necessarily brief and free of extraneous facts that may overload the 
channel. We're not advocating the addition of note-taking to 
listening tests since note-taking ability would mediate the test 
results. We are saying that we have to keep in mind the stimulus 
material. Also, distractions such as those found in many listening 
tests may not be typical of those occurring in everyday situations, 
ihe controversial material discussed in the KCLT distraction 
stimulus material would lead anyone astray. 

2. Listening to an audiotape may very well differ from receiving the 
stimulus via videotape or in actual performance. Viueotape has been 
found to be more involving and learning has been positively related 
to the 2-channel mode mor'i than to a one-channel mode. The use of 
videotape presents problems in mass- testing situations, however. 
And real -life performance of the stimulus material leads to lack of 
control, so it is most important that the material be audible and 
understandable, whatever medium is used. 

We, in the North, found the dialects most distracting and many stu- 
dents were unable to understand what was being said, even though 
they were listening. Listening to dialects is another form of 
distraction, one which would not be typical of most college settings 
(in that students would be more attuned and used to listening to a 
Professor with a dialect). In addition, our copy of the Watson- 
Barker test was most inferior. Dual -channel presentation would have 
helped. 

3. There are problems with all modes of responding on listening tests. 
Speaking ability would mediate the test results when students answer 
orally. With the Watson-Barker and Kentucky tests, reading ability 
is a mediator. We noticed many students who were unable to read all 
the responses in the amount of time given for responding to the 
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questions. Perhaps they had listened, but didn't make it through 
all the responses. Or perhaps they were busy reading when the next 
stimulus item was presented, so the next answer might be incorrect. 

We also found that many students (in addition to the 12 honest ones) 
completed the lecture listening portion of the Kentucky test as they 
were hearing it. It was just too easy for them to turn the page 
over and listen for the answers. We must be aware of the potential 
of these factors, with the multiple-choice version of the CCAI 
test, we didn't hand out the questions until the videotape was 
finished playing (and they had completed the open-ended responses), 
so that reading ability wouldn't interfere with the results. Yet, 
guessing can occur on all multiple choice instruments and we can't 
be sure how much of the listening score is a result of good 
guessing. In addition, completing the open-ended version of the 
CCAI would serve as aiaed recall of information that students then 
used on the multiple choice version. 

4. Lastly, questions must be worded accurately. We noticed that some 
of the KCLT questions asked for more than me answer. This is 
misleading in that only one response is correct (KCLT, Question 
#36). Also, some questions had two correct answers. This could 
cause extra confusion for students who listen carefully and know 
that two responses are possible. 

This study has been most enlightening for us. We've discovered 
several additional concerns that must be addressed in listening test 
construction and development. However, the students in this study were 
helped the most. They received four indices of their listening ability 
and now can make choices about the avenues through which they can 
improve their listening skills. Most students in basic courses do not 
have access to this information. 
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TABLE 1 

DESCRIPTIVE STATISTICS ON THE LISTENING TESTS AND SUB-TESTS 



M r AN MEDIAN MODE STANDARD DEVIATION 



KCLT TOTAL 


36.80 


37 


36 


6.58 


STL 


7.16 


7 


7 


2.12 


STL-R 


8.95 


9 


10 


2.03 


INTER 


5.79 


6 


6 


1.78 


LECT 


7.41 


7 


8 


2.29 


DIST 


7.47 


8 


8 


2.17 


WBLT TOTAL 


59.11 


60 


68 


13.02 


INTER-ST 


5.03 


5 


6 


1.99 


DIALOG 


5.21 


5 


5 


1.58 


SH-LECT 


6.67 


7 


8 


1.85 


EMOT 


7.22 


7 


7 


1.50 


DIRECT 


6.15 


6 


6 


1.88 



CCAI-OE TOTAL 


12.29 


U 


12 


2.89 


CCAI-OE 1 


3.70 


4 


5 


1.46 


CCAI-OE 2 


2.98 


3 


3 


.96 


CCAI-OE 3 


2.87 


3 


3 


1.20 


CCAI-OE 4 


2.73 


3 


1 


1.41 


CCAI-MC TOTAL 


7.14 


7 


8 


1.71 
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TABLE 2 
LISTENING TEST CORRELATIONS 





KCLT-TOTAI 

iVvU 1 Ivl rv- 


URI T-T(1TAI 

NOLI — 1 U 1 nL 


LL/U-Ut IUIAL 


LLAI-MC TOTAL 


SEX 


KCLT TOTAL 


1.00 


.32*** 


.11* 


.07 


.12* 


STL 


.70*** 


.18** 


.10 


-.04 


.07 


STL-R 


.73*** 


.21*** 


.07 


.03 


.10* 


INFER 


.41*** 


,23*** 


-.05 


.06 


.24*** 


LECT 


.51*** 


.23*** 


.17** 


.21*** 


-.02 


DIST 


.81*** 


.16** 


.04 


-.04 


.02 



I TOTAL 


.32*** 


1.00 


.39*** 


.27*** 


.04 


INTER-ST 


.25*** 


.63*** 


.31*** 


.17** 


,06 


DIALOG 


.17** 


.55*** 


.25*** 


.11* 


.15** 


SH-LECT 


.21*** 


.63*** 


.31*** 


t 29*** 


-.06 


EM0T 


.21*** 


.53*** 


.17** 


.20*** 


.04 


DIRECT 


.22*** 


.63*** 


.30*** 


.20*** 


.04 



CCAI-0E TOTAL 


.11* 


.39*** 


1.00 


.35*** 


.12* 


CCAI-OE 1 


.04 


.09 


.49*** 


.05 


.05 


CCAI-OE 2 


-.01 


.16** 


.44*** 


t 27*** 


.12* 


CCAI-OE 3 


.07 


.24** 


.65*** 


t 22*** 


.06 


CCAI-OE 4 


.14* 


.38*** 


.69*** 


.28*** 


.06 


CCAI-MC TOTAL 


.07 


.27*** 


.35*** 


1.00 


-.01 


Note: *p < .05 


**D < .01 


***p < .001 
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